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ABSTRACT 

This study examines the effect of choosing versus receiving feedback on the learning performance of n = 98 
post-secondary students from California on a digital poster design task. The study employs a yoked experimental design 
where college students are randomly assigned to play a choice-based assessment game, Posterlet, in one of two 
conditions, Choose or Receive. In the Choose condition, students choose confirmatory (i.e., positive) or critical (i.e., 
negative) feedback about their posters. In the Receive condition, students are assigned the same feedback valence that 
students in the Choose condition chose. Results show that 1) critical feedback and revision are positively associated with 
learning performance when students choose their feedback, but critical feedback is negatively associated with learning 
performance when students receive their feedback; 2) there is no significant effect of feedback choice between 
conditions; and 3) students enjoy and spend time designing posters significantly more when they choose rather than 
receive their feedback. Implications for designing feedback-rich environments are discussed. 


KEYWORDS 

Feedback, Choice, Assessment, Game, Learning, Performance 


1. INTRODUCTION 

Feedback plays a major role in educational performance, but it yields mixed results within the educational 
literature (Hattie and Timperley, 2007). Several meta-analyses found that, although feedback improved 
performance in general, it had no effect or it hindered performance in a third of the studies examined (Alfieri 
et al., 2011; Bangert-Drowns et al., 1991; Kluger and DeNisi, 1998). Moreover, the role of feedback valence 
(i.e., confirmatory or critical) on performance is still a matter of debate (Fishbach et al., 2010; Gregory and 
Levy, 2015). Furthermore, most research focuses on feedback being assigned to the learner. In contrast, in 
previous research, the impact of feedback choice on performance in an educational setting was investigated 
and results indicated that choosing critical feedback was positively associated with learning and performance 
(Cutumisu et al., 2015). Additionally, in the current study, the effect of feedback agency (i.e., choosing 
versus receiving feedback) on college students’ performance is examined for the first time. This topic has 
wide applicability. For example, in medical research, patients who had control over their level of pain 
medication chose lower doses than those prescribed by medical staff (Haydon et al., 2011). In educational 
research, an open research question of relevance for designing feedback environments is whether students 
who have control over their feedback valence choose more critical feedback and learn more than students 
who receive (i.e., are assigned) feedback. Therefore, this paper starts tackling this question by focusing on 
two orthogonal dimensions of feedback: choice (feedback is chosen or assigned) and valence (feedback is 
confirmatory or critical). 

Specifically, this research aims to examine the effect of feedback agency (i.e., choosing versus receiving 
feedback) on the performance of college students, by comparing learning outcomes between participants who 
choose feedback and those who are assigned the same amount, valence, and order of feedback. In addition to 
performance and time on task, students’ reported enjoyment of designing posters is assessed. An 
experimental study was designed to address the following research questions: 
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1) Does critical feedback correlate with performance outcomes by condition? 

2) Are there outcome differences between choosing and receiving feedback? 

3) Are there enjoyment differences between choosing and receiving feedback? 

The remainder of the paper reviews the related literature, then it describes the experimental study and the 
methods, it presents evidence of the impact of choosing versus receiving feedback on learning outcomes, and 
it concludes with a discussion and implications of this research. 


2. LITERATURE REVIEW 

There are many factors that influence the effectiveness of feedback for performance. For example, recent 
research findings point to factors that include feedback message construction, delivery of critical feedback, 
credibility of the source, specificity and relevance, and feedback orientation (i.e., an individual’s positive 
affect, interest, and engagement with feedback; Anseel et al., 2015; Ashford et al., 2016; De Stobbeleir et al., 
2011; Gregory and Levy, 2015; Landis-Lewis et al., 2015; Porath et al., 2015). 

Critical feedback. Moreover, critical feedback seems to aid performance in some situations (Kluger and 
DeNisi, 1998), with individual factors, such as mindset, affecting engagement with critical feedback 
(Mangels et al., 2006). Although choosing critical feedback is positively associated with learning (Cutumisu 
et al., 2015), it is still not known whether it is the choice over feedback or the actual amount of critical 
feedback that impacts the performance of college students. This suggests that both feedback agency and 
valence are worth exploring for this population in relation with learning and performance. 

Choice-based assessments. A game-based dynamic assessment, Posterlet, was employed to examine the 
impact of students’ choices to seek critical feedback on performance and learning. In addition to 
confirmatory and critical feedback choices, Posterlet offers players opportunities to learn graphic design 
principles while designing a digital poster (Cutumisu et al., 2015). The design of Posterlet draws on 
constructivist, choice-based assessments (Schwartz et al., 2009; Schwartz and Arena, 2013) that emphasize 
learning during the assessment and that shift the assessment focus from the learning outcomes to the learning 
processes (e.g., choosing critical feedback) involved in solving a challenge (e.g., designing a poster). 
Specifically, in Posterlet, players take on the role of designing posters for booths at a fair. They choose a 
booth of interest (e.g., basketball toss) and they design a poster using a graphical user interface provided by 
the game. Posterlet measures two choices that a player makes upon completing a poster: 1) choose either 
confirmatory or critical feedback from three virtual characters about the poster and 2) choose to revise the 
poster after reading all three pieces of feedback. A variation of the Posterlet game was specifically designed 
to enable a performance outcome comparison between college students who choose and those who receive 
(i.e., are assigned) feedback. In this new version, feedback is assigned to the player in a principled way that 
mirrors the feedback chosen by a corresponding player of the original Posterlet version. 


3. METHODS 


3.1 Participants and Procedure 

Participants are n = 98 (55 female) students aged 18 to 52, M age = 22.71 years (SD = 5.59) from a college in 
California. Both versions of the Posterlet game and a post-test were employed to collect data between Spring 
2014 and Spring 2015. All participants provided consent and were randomly assigned to one of two 
conditions, Choose (n = 49 students aged 18 to 52, M age = 22.53 years, SD age = 5.96, 32 female) and Receive 
(n = 49 students aged 18 to 38, M age = 22.90 years, SD age = 5.26, 23 female), according to a yoked study 
design described in a related study (Cutumisu and Schwartz, 2016). These conditions correspond to two 
different game versions. In the Choose game version, students choose their feedback valence, as illustrated in 
Figure la, by selecting either critical or confirmatory feedback from each virtual character. In the Receive 
game version, students are assigned their feedback valence, as illustrated in Figure lb. After designing 
posters in two rounds of the game for M C hoose = 8.72 minutes (SD = 3.28) and M Rece ive = 7.54 minutes 
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(SD = 3.84), all participants completed the same online post-test that measured their knowledge of graphic 
design principles and enjoyment of designing posters. The time students spent on the post-test in each 
condition was M C hoose = 6.56 minutes (SD = 1.44) and M Rece ive = 7.58 minutes (SD = 5.49), respectively. 



a 


( don’t like... 

Where is the Fall Fair 

again? , ... 

I like... Click for feedback 

Wow! You're good at 
arranging words. I can 
read everything. 



b 


Figure 1. In the Choose condition (a), the player chose critical feedback from the lion and then confirmatory feedback 
from the elephant. In the Receive condition (b), the player first received critical feedback from the elephant and then 

confirmatory feedback from the ostrich 


3.2 Measures and Data Sources 

Students completed one of two versions of the game, with the primary difference being whether 1) they chose 
between critical and confirmatory feedback or 2) they were assigned a schedule of critical and confirmatory 
feedback (Cutumisu and Schwartz, 2016). Posterlet measured students’ behaviors and performance in the 
game. A post-test following the game measured students’ knowledge of graphic design principles acquired 
through playing Posterlet. 

Measures of student choices. Critical Feedback measures the number of times that the student chose or 
received critical (i.e., I don ’t like) feedback, ranging from 0 (students chose/received only confirmatory 
feedback across the game) to 6 (students chose/received only critical feedback across the game). Revision 
measures the number of times that the student chose to revise a poster, ranging from 0 (the student did not 
revise any posters) to 2 (the student revised both posters). All students had a choice to revise their posters, 
even though students in the Receive condition did not have a choice regarding the valence of their feedback. 

Measures of student performance. Poster Quality measures the poster performance (i.e., the quality of 
the posters created by the student in Posterlet). The game evaluates each poster against a set of 21 graphic 
design rules provided by a graphic artist. For each poster, the game evaluates each rule with 1 , if the rule is 
always used correctly on that poster; 0, if the rule is not applicable on that poster; and -1, if the rule is used 
incorrectly on that poster. The score of any individual poster created by a student represents the sum of all 21 
rule scores, ranging from -21 to 21. Thus, Poster Quality represents the score sum across the game of the last 
individual poster on each round, ranging from -42 to 42. 

Measures of time on task. Design Duration measures the amount of time (in minutes) students spent 
designing all posters, including revisions. Specifically, the game starts measuring the time a student spends 
from choosing a poster theme (e.g., basketball) until submitting the first poster draft. If the student chooses to 
revise the poster after reading the feedback, then this measure includes the additional time that the student 
spends updating that poster. 

Measures of student learning. Poster Ranking measures a student’s knowledge of design principles on a 
post-test independent of the game. After completing the game, the student is directed to an online post-test to 
assess four sets of posters. Each set contains two versions of a poster, featuring a design principle used 
correctly on one poster and incorrectly on the other poster. For each set, the student is shown in a five-second 
succession the first poster, a pattern image, and a modified version of the first poster, as illustrated in Figure 
2. Then, the student decides whether the second poster is the same, better, or worse than the first poster. Each 
answer is scored with 1, if it is correct and 0, if it is incorrect. Thus, Poster Ranking ranges from 0 to 4. 
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Measures of Enjoyment. Enjoyment measures the enjoyment of designing posters on a 1-5 Likert-type 
response scale, where 1 = none and 5 = a huge amount. Students answered this question after completing the 
game and the post-test. 


FIRST PATTERN LAST 

POSTER POSTER 



Is the LAST poster: 

o the same as the first poster? 
© better than the first poster? 
worse than the first poster? 


Figure 2. One of the four post-test items measuring students’ knowledge of graphic design principles. This item’s target 
feature is the relevance of graphics (a basket ball instead of a soccer ball) for the poster’s theme (basketball) 


4. RESULTS 


4.1 Does Critical Feedback Correlate with Performance Outcomes by 
Condition? 

Spearman’s rank correlation coefficient ( rho ) is reported in all the analyses included in this section, because 
the variables measured were not normally distributed. In the Choose condition, Critical Feedback correlates 
with performance on the posters measured by Poster Quality and strongly with Revision, as shown in Table 
1. The in-game poster performance (Poster Quality) can be considered to be a learning measure, because 
students improved their performance from the first to the second game round. Poster 1 is considered to be the 
pretest, being the first poster designed by the player in the Posterlet game, before revision. Poster 2 is the last 
poster designed by the student in the Posterlet game, after a potential revision. A repeated measures ANOVA 
analysis reveals that the poster quality of the students in the Choose condition increased significantly from 
Poster 1 (M = 9.59, SD = 5.69) to Poster 2 (M = 13.04, SD = 3.96); F(l, 48) = 24.67, p < .001, partial eta 
squared = .34. As well, in the Receive condition, poster quality increased significantly from Poster 1 (M = 
9.33, SD = 6.93) to Poster 2 (M = 12.63, SD = 4.37); F(l, 48) = 14.72, p < .001, partial eta squared = .23. 
Critical Feedback also correlates positively with Poster Ranking, although not statistically significantly. 


Table 1. Correlations between choices and performance outcomes (in-game and post-test) in the Choose condition 


Measures (n = 49) 

Revision 

Poster Quality 

Poster Ranking 

Critical Feedback 

.64** 

.28* 

.22 

Revision 

- 

.27 

.14 

Poster Quality 

- 

- 

.18 


**/?<. 01 * p < .05 


In the Receive condition, Critical Feedback correlates strongly with Revision and inversely with poster 
quality, as shown in Table 2. Although not statistically significantly, Critical Feedback also inversely 
correlates with performance on the graphic design principles measured by Poster Ranking in this condition. 

Table 2. Correlations between behaviors and performance outcomes (in-game and post-test) in the Receive condition 


Measures (n = 49) 

Revision 

Poster Quality 

Poster Ranking 

Critical Feedback 

44 ** 

-.16 

-.34* 

Revision 

- 

.09 

.10 

Poster Quality 

- 

- 

.23 


** p < .01 

* p < .05 
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Taken together, these results imply that critical feedback correlates positively with learning outcomes 
only when students exercise a feedback choice, not when they are assigned their feedback. In support of this 
conclusion, results show that the relation between critical feedback and the measures of learning differs 
statistically significantly between the two conditions. Specifically, a t-test analysis comparing the correlation 
coefficients of Critical Feedback and Poster Quality between the Choose and Receive conditions indicated 
that the two correlation coefficients were statistically significantly different from each other ( z-score = 2.15, 
p = .03; Fisher, 1921; Soper, 2016). A comparison of the Critical Feedback and Poster Ranking correlation 
coefficients between conditions yielded similar results (z-score = 2.77, p < .01). 


4.2 Are there Outcome Differences between Choosing and Receiving 
Feedback? 

Revision. An independent- samples t-test was conducted to compare Revision between students in the two 
conditions. There were no significant differences in Revision between students in the Choose (M = .76, 
SD = .83) and Receive condition (M = .53, SD = .62); f(88.56) = 1.52, p = .13. Figure 3a shows the mean 
Revision plotted by number of Critical Feedbacks (0-6) for each of the two conditions. 




Figure 3. Mean Revision (a) and mean Poster Quality (b) by critical feedback and condition 

Poster Quality. An independent- samples t-test was conducted to compare Poster Quality between 
conditions. There were no significant differences in Poster Quality between students in the Choose 
(M = 23.57, SD = 8.85) and Receive condition (M = 23.51, SD = 9.48); t( 96) = .03, p = .97. Figure 3b shows 
the mean Poster Quality plotted by Critical Feedback for each of the two conditions. Another independent- 
samples t-test was conducted to compare the quality of the Pretest (the first poster before revisions) between 
students in both conditions. There were no significant differences in Pretest between students in the Choose 
(M = 9.59, SD = 5.69) and Receive condition (M = 9.33, SD = 6.93); t( 96) = .21, p = .84. 

Poster Ranking. An independent- samples t-test was conducted to compare Poster Ranking between 
students in the two conditions. There were no significant differences in Poster Ranking between students in 
the Choose (M = 1.71, SD = 1.02) and Receive condition (M = 1.63, SD = 1.07); t( 96) = .39, p = .70. Figure 
4a shows the mean Poster Ranking plotted by Critical Feedback for each of the two conditions. 




Figure 4. Mean Poster Ranking (a) and mean Design Duration (b) by critical feedback and condition 
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Design Duration. An independent- samples t-test was conducted to compare Design Duration between 
conditions. There were no significant differences in Design Duration between students in the Choose 
(M = 8.72 minutes, SD = 3.28) and students in the Receive condition (M = 7.54 minutes, SD = 3.84); 
t( 96) = 1.63, p = .1 1. Figure 4b shows the mean Design Duration plotted by Critical Feedback. 

4.3 Are there Enjoyment Differences between Choosing and Receiving 
Feedback? 

An independent-samples t-test was conducted to compare Enjoyment between conditions. Results show that 
students in the Choose condition (M = 4.03, SD = .80) enjoyed designing posters significantly more than 
students in the Receive condition (M = 3.45, SD = .91); f(81) = 2.99, p < .01. Figure 5 shows the mean 
Enjoyment plotted by critical feedback for each condition, indicating that enjoyment is hardly affected by the 
amount of critical feedback, except at the high end. These results suggest that the effect on enjoyment could 
be simply the existence of choice, rather than the actual valence of the choice. In the Choose condition, 76% 
of the students reported 4 and 5 levels of enjoyment, as opposed to only 51% of the students in the Receive 
condition. Moreover, Enjoyment is positively associated with Design Duration ( rho = .37, p = .03) only in 
the Choose condition. 



Figure 5. Mean Enjoyment by critical feedback and condition 


5. DISCUSSION AND IMPLICATIONS 

Critical feedback and performance outcomes. Critical Feedback and Revision were strongly correlated 
and students improved their poster design performance as they played the game. The more the students chose 
critical feedback and revised, the better they performed on the poster design task. This is consistent with 
research reporting that giving children a choice led to better information- seeking performance in a text search 
task (Reynolds and Symons, 2001). Concomitantly, the more the students were assigned critical feedback, the 
worse they performed on the post-test. However, the graphical representation of poster ranking by critical 
feedback per condition suggests that critical feedback could aid performance depending on the amount of 
critical feedback assigned. A limitation of this study is the reduced amount of data points for each critical 
feedback value. More data will be collected to better understand how students’ performance relates to critical 
feedback. Overall, critical feedback is associated with better performance when students choose their 
feedback valence, but with worse performance when they are assigned their feedback valence. One possible 
explanation for this outcome is that choice can be a source of motivation, which may lead students to engage 
more with their learning. For example, high-school students who chose which of their homework 
assignments to complete outperformed their peers who had no choice available (Patall et al., 2010). 
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Outcome differences between choosing and receiving feedback. This research hypothesized that 
students who chose feedback performed better than students who were assigned feedback. However, results 
revealed no effect of feedback choice, consistent with a related prior study examining Mechanical Turk 
adults (Cutumisu and Schwartz, 2016). There were no differences between conditions in the choice to revise, 
performance on the posters and on the post-test, and time spent designing posters. This indicates that no 
underlying variable (e.g., mindset) drives the effect of critical feedback, since assigning the same amount of 
feedback leads to the same results as other factors that may cause students to choose critical feedback. A 
limitation of this study could serve as an explanation for this outcome: both game versions consisted of two 
rounds and five minutes for poster design. Future research will include one more round in each condition and 
provide more time per round to assess students’ performance in both conditions and perhaps note differences. 

Enjoyment differences between choosing and receiving feedback. Students in the Choose condition 
enjoyed designing posters significantly more than those in the Receive condition. Choices can promote a 
sense of autonomy that could be highly motivating for the students (Deci and Ryan, 1985). Although greater 
perceived autonomy is associated with higher levels of enjoyment and intrinsic motivation (Reeve et al., 
1999), choice is not always a motivator (Katz and Assor, 2007). The finding that enjoyment correlates with 
the time the students took to design the posters only in the Choose condition supports this motivational 
hypothesis and is consistent with prior research involving a sample of Mechanical Turk adults (Cutumisu and 
Schwartz, 2016). The majority of the students reported high levels of enjoyment, which is encouraging for a 
game that is ultimately an assessment. Thus, games such as Posterlet could be enjoyable assessment 
environments for college students, especially when they provide feedback valence choices. 

Educational implications. One lesson that can be taken away from this study is that choice of feedback 
seems to play an important role for performance, revision, time on task, and enjoyment for college students. 
Thus, feedback choice in general, and feedback choice between confirmatory and critical feedback in 
particular, should be considered as an important feature in instructional environments for this population to 
maximize feedback’s effectiveness in impacting performance and learning. A feedback choice-rich learning 
and assessment environment could also lead to student enjoyment and more time spent on task. For example, 
in a physical activity intervention drawing on self-determination theory (Deci and Ryan, 1985), students who 
were taught by autonomy- supportive teachers participated more frequently in leisure-time physical activities 
than students who were taught by less autonomy- supportive teachers (Chatzisarantis and Hagger, 2009). 
Even though no significant differences in outcomes were found between choosing and receiving feedback, 
engaging with a certain threshold of critical feedback may make a difference for learning. Thus, existing 
assessment environments that do not provide choices could integrate dynamic ways of adjusting the amount 
of critical feedback available to students and evaluate their performance in each case. 


6. CONCLUSION 

The study presented a preliminary comparison between the impact of choosing and receiving feedback on 
college students’ performance. Critical feedback and performance are associated positively when students 
choose their feedback, but negatively when they are assigned their feedback. The choice to revise is 
positively associated with performance, but only when students choose their feedback. Students who choose 
their feedback enjoy designing posters significantly more than students who receive their feedback. 
Moreover, students who have a choice regarding their feedback also spend significantly more time designing 
posters than students who are assigned their feedback. Future research in this area needs to address the 
theoretical underpinnings of feedback valence and choice, and further examine the impact of motivation, 
enjoyment, and critical feedback on students’ performance and learning. 
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